6. Inverse theory methods

6.1 Review of the least-squares fitting

Linear regression that we used for the last project, was essentially a problem of minimizing the following sum of squares of residuals, i.e. the differences between the data $(x_i,y_i)$ and the linear model $y(x)=c_0+c_1 x$ predictions: $$ \min \left[ \sum_{i=1}^m r_i^2 \right] , \quad r_i = y_i - (c_0 + c_1 x_i) $$ In matrix form, the problem is to solve the set of linear equations $$ \vec{y} = \left( \begin{array}{c} y_1 \\ y_2 \\ \vdots \\ y_m\end{array} \right) = \left( \begin{array}{cc} 1 & x_1 \\ 1 & x_2 \\ \vdots & \dots \\ 1 & x_m\end{array} \right) \left( \begin{array}{c} c_0 \\ c_1 \end{array} \right) $$ for $\{ c_0,c_1\}$. In octave this can be accomplished by using operator \ :

There is another way to look at the solution of this overdetermined problem $$ \vec{y} = {A} \vec{c} $$ Any full rank matrix $A ∈ R^{m×n}$ , with $m \geq n$, admits a unique $QR$ factorization $A = QR$. It is possible to prove that $A = \tilde{Q}\tilde{R}$ where $\tilde{Q}=Q(1:m.1:n)$ and $\tilde{R}=R(1:n,1:n)$ are the submatrices as showni below (figure from A.Quarteroni, F.Saleri and P.Gervasio, Scientific Computing with MATLAB and Octave, 3rd ed, Springer).

[Fig5.9]

$\tilde{Q}$ has orthonormal column vectors, while $\tilde{R}$ is a non-singular upper triangular matrix. Therefore, the unique solution is given by $$ \vec{c}^* = (\tilde{R})^{-1} (\tilde{Q})^T \vec{y} $$

But what if the task is not to minimize the (vertical) differences between $y_i$ and $c_0 + c_1 x_i$, but instead the sum of the distances between the data points and the line of fit? Another formulation of the problem is the "minimum length of water pipes" of an example distributed with eXtrema, see https://www.physics.brocku.ca/Labs/extrema/.

On a plane, a straight line can be represented by the equation $$ c_0 + c_1 x + c_2 y = 0, \quad c_1^2 + c_2^2 = 1 $$ where $(c_1,c_2)$ is the unit vector perpendicular to the line. For points $(x_i,y_i)$ not on the line, the above equation represents the distances of the points from the line, $$ r_i = c_0 + c_1 x_i + c_2 y_i $$ and the problem becomes the minimization of the total sum of such distances: $$ \min ||\vec{r} ||^2 = \min \left[ \sum_{i=1}^m r_i^2 \right] $$ subject to $$ \vec{r} = \left( \begin{array}{c} r_1 \\ r_2 \\ \vdots \\ r_m\end{array} \right) = \left( \begin{array}{ccc} 1 & x_1 & y_1 \\ 1 & x_2 & y_2 \\ \vdots & \dots & \vdots\\ 1 & x_m & y_m\end{array} \right) \left( \begin{array}{c} c_0 \\ c_1 \\ c_2\end{array} \right) = A \vec{c} \quad \mbox{and} \quad c_1^2+c_2^2=1 $$

Commands qr(), triu(), and svd() will come handy.

6.2 Introduction to inverse problems

Inverse Theory presentation (pdf)

6.3 Homework, due 2022-11-14

[pmma.png]